The present invention relates to vehicle control systems in vehicles, Specifically, the present invention detects and classifies objects in real time, e.g. oncoming vehicle headlights, leading vehicle taillights and streetlights, in a series of images obtained from a camera mounted on a vehicle, The images are used in parallel by a number of vehicle control systems including lane departure detection, forward collision control and headlight control systems. The classification of objects is preferably used by more than one of the vehicle control systems. In a headlight control system, the classification of objects is used to provide a signal for switching the headlights between high beams and low beams.
Headlight high beams are a distraction and create a safety hazard by blinding the driver of an oncoming vehicle or leading vehicle. It is not uncommon for a driver to forget to lower the high beams, thus creating a safety hazard for another driver. It is thus desirable to automatically control the state of a vehicle's headlights. Automatic vehicle headlight control also increases the use of high beams in conditions which allow their use, increasing the safety as well as reducing the hazard caused by the occasional failure of the driver to deactivate the high beams which distract the other driver.
Prior art control systems which automatically control the vehicle headlights have included a single light sensor which integrates light in the scene forward of the vehicle. When the integrated light exceeds a threshold, the vehicle headlights are dimmed.
Vehicle headlight control using cameras has also been described. In a system, as disclosed by Schofield et al. (U.S. Pat. No. 6,831,261), a headlight control device is capable of identifying characteristics of light sources is based upon an evaluation of light source characteristics in the scene forward of the vehicle. In the vicinity of each light source, each light source is separated from the remainder of the scene and analyzed to determine characteristics of the light source. One characteristic used to identify a light source is the spectral characteristic which is compared with spectral signatures of known light sources, such as those of automobile headlights and taillights. Another characteristic used in identifying a light source is the spatial layout of the light source. By providing the ability to identify the headlights of oncoming vehicles and the tail lights of leading vehicles, the state of the headlights of the controlled vehicle may be adjusted in response to the presence or absence of either of these light sources or the intensity of these light sources. In order to respond to the different characteristics of headlights and tail lights, a different exposure period is provided for the array in order to detect and identify each light source. In particular, the exposure period may be longer for detecting leading taillights and significantly shorter for detecting oncoming headlights. A solid-state light imaging array is provided that is made up of sensors arranged in a matrix on at least one semiconductor substrate. The light-imaging array includes a_spectral separation device, and each of the sensors responds to light in a particular spectral region. The control circuit responds to the sensors in order to determine if spatially adjacent regions of the field of view forward of the vehicle include light of a particular spectral signature above a particular intensity level. In this manner, the control identifies light sources that are either oncoming headlights or leading taillights by identifying such light sources according to their spectral makeup. Spatial evaluation may be implemented by selecting characteristics of an optical device provided with the imaging sensor, such as providing increased magnification central of the forward scene, or providing a wide horizontal view and narrow vertical view.
In the vehicle headlight control system, as disclosed in U.S. Pat. No. 6,831,261 special controls are required for camera settings including exposure time and magnification, for instance multiple exposures each with different exposure times.
Reference is now made to FIGS. 1 and 1a (prior art) which illustrate a vehicle control system 16 including a camera or image sensor 12 mounted in a vehicle 18 imaging a field of view in the forward direction. Image sensor 12 typically delivers images in real time and the images are captured in a time series of image frames 15. An image processor 14 is used to process image frames 15 to perform a number of prior art vehicle controls.
Exemplary Prior Art Vehicle Controls are:
Step 17-Collision Warning is disclosed in U.S. Pat. No. 7,113,867 by Stein, and included herein by reference for all purposes as if entirely set forth herein. Time to collision is determined based on information from multiple images 15 captured in real time using camera 12 mounted in vehicle 18.
Step 19-Lane Departure Warning (LDW), as disclosed in U.S. Pat. No. 6,882,287 by Scofield. If a moving vehicle has inadvertently moved out of its lane of travel based on image information from images 15 from forward looking camera 12, then system 16 signals the driver accordingly.
Step 21-Ego-motion estimation is disclosed in U.S. Pat. No. 6,704,621 by Stein and included herein by reference for all purposes as if entirely set forth herein, Image information is received from images 15 recorded as the vehicle moves along a roadway. The image information is processed to generate an ego-motion estimate of the vehicle, including the translation of the vehicle in the forward direction and the rotation. Vehicle control systems, such as disclosed in U.S. Pat. No. 6,831,261 which rely on changing exposure parameters (ie, aperture, exposure, magnification, etc) in order to detect headlights have a difficult time maintaining other control systems which rely on the same camera, e.g. Lane Departure Warning, Forward Collision Warning, etc. As a result of changing exposure parameters half or more of the (possibly critical) frames may not be available for the other control systems. This greatly affects performance of the other control systems.
Hence, since in the vehicle headlight control system as disclosed in U.S. Pat. No. 6,831,261 (or in any other disclosure where special control is required of camera settings including, aperture, exposure time and magnification), the same camera cannot be conveniently used for other simultaneously operable vehicle control systems such as LDW 19 or collision warning 17.
Additionally, the use of color cameras with infrared filters required to achieve good spectral separation reduces imaging sensitivity by a factor of six or more. A reduction in sensitivity by such a factor has an adverse impact on other vehicle control application such as LDW performance in dark scenes. The presence of an infrared filter also negates the use of the camera as a near infrared sensor for applications, such as pedestrian detection. Thus, headlight control systems which make strong use of color or spectral analysis in the captured images (such as in U.S. Pat. No. 6,831,261) will tend not be compatible with other applications without sacrificing performance.
There is thus a need for, and it would be highly advantageous to have a method of detecting and classifying objects in real time, e.g. oncoming vehicle headlights, leading vehicle taillights and streetlights, in a series of image frames 15 obtained from a camera mounted in a vehicle to provide a signal, specifically with image frames 15 available for use by a number of vehicle control applications.
The term “classification” as used herein refers to classifying the object of which a spot in an image frame is the image of a real object, e.g. headlight, tail lights. The term “classification” is used to refer interchangeably to the spot or to the object.
Radial basis functions (RBF) are used for interpolation in a stream of data. Radial basis functions (RBF) differ from statistical approaches in that approximations must be performed on streams of data rather than on complete data sets. RBFs use supervised learning and sometimes unsupervised learning to minimize approximation error in a stream of data. They are used in function approximation, time series prediction, and control. http://en.wikipedia.org/wiki/Radial_basis_function
The terms referring to image space such as “downward”, “inward” “upward”, “upper”, “lower”, “bottom and “top” refer to a non-inverted image as viewed on a monitor from camera 12.